System with 3D user interface integration

ABSTRACT

Disclosed is a system comprising a handheld device and at least one display. The handheld device is adapted for performing at least one action in a physical 3D environment; wherein the at least one display is adapted for visually representing the physical 3D environment; and where the handheld device is adapted for remotely controlling the view with which the 3D environment is represented on the display.

FIELD OF THE INVENTION

This invention generally relates to a method and a system comprising ahandheld device and at least one display.

BACKGROUND OF THE INVENTION

3D visualization is important in many fields of industry and medicine,where 3D information is becoming more and more predominant.

Displaying and inspecting 3D information is inherently difficult. Tofully understand a 3D object or entire environment on a screen, the usershould generally be able to rotate the object or scene, such that manyor preferentially all surfaces are displayed. This is true even for 3Ddisplays, e.g. stereoscopic or holographic, where from a given viewingposition and with a given viewing angle, the user will only see somesurfaces of an arbitrary 3D environment. Often, the user will also wantto zoom into details or zoom out for an overview.

Various user interaction devices are in use for software that displays3D data; these devices are: 3D mice, space balls, and touch screens. Theoperation of these current interaction devices requires physicallytouching them.

Physically touching a user-interaction device can be a disadvantage inmedical applications due to risks of cross-contamination betweenpatients or between patient and operator, or in industrial applicationsin dirty environments.

Several non-touch user interfaces for 3D data viewing in medicalapplications have been described in the literature. Vogt et al (2004)describe a touchless interactive system for in-situ visualization of 3Dmedical imaging data. The user interface is based on tracking ofreflective markers, where a camera is mounted on the physician's head.Graetzel et al (2004) describe a touchless system that interprets handgestures as mouse actions. It is based on stereo vision and intended foruse in minimally invasive surgery.

It remains a problem to improve systems that require user interfaces forview control, which for example can be used for clinical purposes.

SUMMARY

Disclosed is a system comprising a handheld device and at least onedisplay, where the handheld device is adapted for performing at leastone action in a physical 3D environment, where the at least one displayis adapted for visually representing the physical 3D environment, andwhere the handheld device is adapted for remotely controlling the viewwith which said 3D environment is represented on the display.

The system may be adapted for switching between performing the at leastone action in the physical 3D environment, and remotely controlling theview with which the 3D environment is represented on the display.

The system disclosed here performs the integration of 3D user interfacefunctionality with any other handheld device with other operatingfunctionality, such that the operator ideally only touches this latterdevice that is intended to be touched. A particular example of such ahandheld device is one that records some 3D geometry, for example ahandheld 3D scanner.

The handheld device is a multi-purpose device, such as a dual-purpose ortwo-purpose device, i.e. a device both for performing actions in thephysical 3D environment, such as measuring and manipulating, and forremotely controlling the view of the 3D environment on the display.

Geometrically, a view is determined by the virtual observer's/camera'sposition and orientation relative to the 3D environment or its visualrepresentation. If the display is two-dimensional, the view is alsodetermined by the type of projection. A view may also be determined by amagnification factor.

The virtual observer's and the 3D environment's position and orientationare always relative to each other. In terms of user experience insoftware systems with 3D input devices, the user may feel that forexample, he/she is moving the 3D environment while remaining stationaryhimself/herself, but there is always an equivalent movement of thevirtual observer/camera that gives the same results on the display.Often, descriptions of 3D software systems use the expression “pan” toindicate an apparent translational movement of the 3D environment,“rotate” to indicate a rotational movement of the 3D environment, and“zoom” to indicate a change in magnification factor.

Graphically, a view can represent a 3D environment by means ofphotographs or as some kind of virtual representation such as a computergraphic, or similar. A computer graphic can be rendered for example withtexture and/or shading and/or virtual light sources and/or light modelsfor surface properties. A computer graphic can also be a simplifiedrepresentation of the 3D environment, for example a mesh, an outline, oran otherwise simplified representation. All or parts of the 3Denvironment can also be rendered with some degree of transparency. Aview may represent the 3D environment in total or only parts thereof.

All of the touch-less prior art systems are 3D user interface devicesonly. In many prior art applications, the operator using such userinterface device will also hold and work with another device that reallyis the central device in the overall application, e.g. a medicalinstrument.

It is thus an advantage of the present system that the 3D user-interfacefunctionality is integrated in the central device, which is used forperforming some kind of action.

In some embodiments the handheld device is adapted for remotelycontrolling the magnification with which the 3D environment isrepresented on the display.

In some embodiments the handheld device is adapted for changing therendering of the 3D environment on the display.

In some embodiments the view is defined as viewing angle and/or viewingposition.

In some embodiments the at least one action comprises one or more of theactions of:

-   -   measuring,    -   recording,    -   scanning,    -   manipulating,    -   modifying.

In some embodiments the 3D environment comprises one or more 3D objects.

In some embodiments the handheld device is adapted to be held in onehand by an operator.

In some embodiments the display is adapted to represent the 3Denvironment from multiple views.

In some embodiments the display is adapted to represent the 3Denvironment from different viewing angles and/or viewing positions.

In some embodiments the view of the 3D environment in the at least onedisplay is at least partly determined by the motion of the operator'shand holding said device.

In some embodiments the magnification represented in the at least onedisplay is at least partly determined by the motion of the operator'shand holding said device.

In some embodiments the handheld device is adapted to record the 3Dgeometry of the 3D environment.

Thus the handheld device may be an intraoral dental scanner, whichrecords the 3D geometry of a patient's teeth. The operator may move thescanner along the teeth of the patient for capturing the 3D geometry ofthe relevant teeth, e.g. all teeth. The scanner may comprise motionsensors for taking the movement of the scanner into account whilecreating the 3D model of the scanned teeth.

The 3D model of the teeth may be shown on a display, and the display mayfor example be a PC screen and/or the like.

The user interface functionality may comprise incorporating motionsensors in the scanner to provide that the user can determine the viewon the screen by moving the scanner. Pointing the scanner down canprovide that the scanned teeth are shown given a downward viewing angle.Holding the scanner in a horizontal position can provide that theviewing angle is likewise horizontal.

In some embodiments the handheld device comprises at least oneuser-interface element. A user-interface element is an element which theuser may manipulate in order to activate a function on the userinterface of the software. Typically the use interface is graphicallypresented on the display of the system.

The handheld device may furthermore be provided with an actuator, whichswitches the handheld device between performing the at least one actionand remotely controlling the view. By providing such a manual switchingfunction that enables the operator to switch between performing the atleast one action and remotely controlling the view, the operator mayeasily control what is performed.

Such an actuator can for example be in the form of a button, switch orcontact. In other embodiments it could be a touch sensitive surface orelement.

In another embodiment the actuator could be a motion sensor provided inthe handheld device that function as the actuator when it registers aspecific type of movement, for example if the operator shakes thehandheld device. Examples of such motion sensors will be describedherein with respect to the user-interface element, however, the personskilled in the art will based on the disclosure herein understand thatsuch motion sensors may also be used as actuators as discussed.

For example, the handheld device can in one embodiment be an intra-oral3D scanner used by a dentist. The scanner is set to be performing theaction of scanning a dental area when the actuator is in one position.When the actuator is switched into a second position the handheld is setto control the view with which the 3D environment is represented on thedisplay. This could for example be that when the dentist have scanned apart of or the complete desired area of an dental arch he can activatethe actuator which then allows the dentist to remotely control the viewof the 3D representation of the scanned area on the display by using thehandheld device.

For example, the actuator could be a button. When the button is pressedquickly the handheld device is prepared for scanning, i.e. it is set forperforming at least one action, the scanning procedure, in the physical3D environment. The scanning is stopped when the button is pressedquickly a second time.

While the scanning is performed a virtual 3D representation is visuallybuilt on the display.

The user can now press and hold the button. This will put the handheldin a controller mode, where the handheld device is adapted for remotelycontrolling the view with which the 3D environment, such as scannedteeth, is represented on the display. While holding the button pressedthe system will use signals from a motion sensor in the handheld deviceto determine how to present the view of the virtual 3D environment.Thus, if the user turns or otherwise moves the hand that holds thehandheld device the view of the virtual 3D environment on the displaywill change accordingly.

Thus, the dentist may use the same handheld device for both scanning anarea and subsequently verifying that the scan has been executedcorrectly without having to move away from the patient or touching anyother equipment than already present in his hands.

In one embodiment the user-interface element is the same as theactuator, or where several user-interface elements are present at leastone also functions as an actuator.

The system may be equipped with a button as an additional elementproviding the user-interface functionality.

In an example the handheld device is a handheld intraoral scanner, andthe display is a computer screen. The operator or user may be a dentist,an assistant and/or the like. The operation functionality of the devicemay be to record some intraoral 3D geometry, and the user interfacefunctionality may be to rotate, pan, and zoom the scanned data on thecomputer screen.

In some embodiments the at least one user-interface element is at leastone motion sensor.

Thus the integration of the user interface functionality in the devicemay be provided by motion sensors, which can be accelerometers insidethe scanner, whose readings determine the orientation of the display onthe screen of the 3D model of the teeth acquired by the scanner.Additional functionality, e.g. to start/stop scanning, may be providedby a button. The button may be located where the operator's or user'sindex finger can reach it conveniently.

Prior art intraoral scanners use a touch screen, a trackball, or a mouseto determine the view in the display. These prior art user interfacedevices can be inconvenient, awkward and difficult to use, and they canbe labor-intensive, and thus costly to sterilize or disinfect. Anintraoral scanner should always be disinfected between scanningdifferent patients, because the scanner is in and may come in contactwith the mouth or other parts of the patient being scanned.

The operator or user, e.g. dentist, may use one hand or both hands tohold the intraoral scanner while scanning, and the scanner may be lightenough and comfortable to be held with just one hand for a longer timewhile scanning.

The device can also be held with one or two hands, while using thedevice as remote control for e.g. changing the view in the display. Itis an advantage of the touchless user interface functionality that inclinical situations, the operator can maintain both hands clean,disinfected, or even sterile.

An advantage of the system is that it allows an iterative process ofworking in a 3D environment without releasing the handheld device duringsaid process. For the above intraoral scanning system example, theoperator, e.g. dentist, can record some teeth surface geometry with ahandheld device that is an intraoral scanner, inspect coverage of thesurface recording by using that same handheld device to move, e.g.rotate, the recorded surface on the display, e.g. a computer screen,detect possible gaps or holes in the coverage of the scanned teeth, andthen for example arrange the scanner in the region where the gaps werelocated and continue recording teeth surface geometry there. Over thisentire iterative cycle, which can be repeated more than once, such as asmany times as required for obtaining a desired scan coverage of theteeth, the dentist does not have to lay the handheld intraoral scannerout of his or her hands.

In some embodiments, the 3D user interface functionality is exploited ina separate location than the operation functionality. For the aboveintraoral scanning system example, the scanning operation is performedin the oral cavity of the patient, while the user interfacefunctionality is more flexibly exploited when the scanner is outside thepatient's mouth. The key characteristic and advantage of the system,again, is that the dentist can exploit the dual and integratedfunctionality, that is operation and user interface, of the scannerwithout laying it out of his or her hands.

The above intraoral scanning system is an example of an embodiment.Other examples for operation functionality or performing actions couldbe drilling, welding, grinding, cutting, soldering, photographing,filming, measuring, executing some surgical procedure etc.

The display of the system can be a 2D computer screen, a 3D display thatprojects stereoscopic image pairs, a volumetric display creating a 3Deffect, such as a swept-volume display, a static volume display, aparallax barrier display, a holographic display etc. Even with a 3Ddisplay, the operator has only one viewing position and viewing anglerelative to the 3D environment at a time. The operator can move his/herhead to assume another viewing position and/or viewing angle physically,but generally, it may be more convenient to use the handheld device withits built-in user interface functionality, e.g. the remote controlling,to change the viewing position and/or viewing angle represented in thedisplay.

In some embodiments the system comprises multiple displays, or one ormore displays that are divided into regions. For example, severalsub-windows on a PC screen can represent different views of the 3Denvironment. The handheld device can be used to change the view in allof them, or only some of them.

In some embodiments the user interface functionality comprises the useof gestures.

Gestures made by e.g. the operator can be used to change, shift ortoggle between sub-windows, and the user-interface functionality can belimited to an active sub-window or one of several displays.

In some embodiments the gestures are adapted to be detected by the atleast one motion sensor. Gestures can alternatively and/or additionallybe detected by range sensors or other sensors that record body motion.

The operator does not have to constantly watch the at least one displayof the system. In many applications, the operator will shift betweenviewing and possible manipulating the display and performing anotheroperation with the handheld device. Thus it is an advantage that theoperator does not have to touch other user interface devices. However,in some cases it may not be possible for the operator to fully avoidtouching other devices, and in these cases it is an advantage that fewertouches are required compared to a system where a handheld device doesnot provide any user interface functionality at all.

In some embodiments the at least one display is arranged separate fromthe handheld device.

In some embodiments the at least one display is defined as a firstdisplay, and where the system further comprises a second display.

In some embodiments the second display is arranged on the handhelddevice.

In some embodiments the second display is arranged on the handhelddevice in a position such that the display is adapted to be viewed bythe operator, while the operator is operating the handheld device.

In some embodiments the second display indicates where the handhelddevice is positioned relative to the 3D environment.

In some embodiments the first display and/or the second display providesinstructions for the operator.

The display(s) can be arranged in multiple ways. For example, they canbe mounted on a wall, placed on some sort of stand or a cart, placed ona rack or desk, or other.

In some embodiments at least one display is mounted on the deviceitself. It can be advantageous to have a display on the device itselfbecause with such an arrangement, the operator's eyes need not focusalternatingly between different distances. In some cases, the operatingfunctionality may require a close look at the device and the vicinity ofthe 3D environment it operates in, and this may be at a distance at mostas far away as the operator's hand. Especially in crowded environmentssuch as dentist's clinics, surgical operation theatres, or industrialworkplaces, it may be difficult to place an external display closely tothe device.

In some embodiments visual information is provided to the operator onone or more means other than the first display.

In some embodiments audible information to the operator is provided tothe operator.

Thus in some embodiments, the system provides additional information tothe operator. In some embodiments, the system includes other visualclues shown on means other than the display(s), such as LEDs on thedevice. In some embodiments, the system provides audible information tothe operator, for example by different sounds and/or by speech.

Said information provided to the operator can comprise instructions foruse, warnings, and the like.

The information can aid with improving the action performance oroperation functionality of the device, for example by indicating howwell an action or operation is being performed, and/or instructions tothe operator aimed at improving the ease of the action or operationand/or the quality of the action or operation's results. For example, aLED can change in color and/or flashing frequency. In a scanner, theinformation can relate to how well the scanned 3D environment is infocus and/or to scan quality and/or to scan coverage. The informationcan comprise instructions on how best to position the scanner such as toattain good scan quality and/or scan coverage. The instructions can beused for planning and/or performing bracket placement. The instructionscan be in the form of a messenger system to the operator.

In some embodiments, some 3D user interface functionality is provided byat least one motion sensor built into the device. Examples of motionsensors are accelerometers, gyros, and magnetometers and/or the like.These sensors can sense rotations, lateral motion, and/or combinationsthereof. Other motion sensors use infrared sensing. For example, atleast one infrared sensor can be mounted on the device and at least oneinfrared emitter can be mounted in the surroundings of the device.Conversely, the at least one emitter can be mounted on the device, andthe at least one sensors in the surroundings. Yet another possibility isto use infrared reflector(s) on the device and both sensor(s) andemitter(s) on the surroundings, or again conversely. Thus motion can besensed by a variety of principles.

Through proper signal processing, some sensors can recognize additionaloperator actions; for example gestures such as taps, waving, or shakingof the handheld device. Thus, these gestures can also be exploited inthe 3D user interface functionality.

In some embodiments the handheld device comprises at least two motionsensors providing sensor fusion. Sensor fusion can be used to achieve abetter motion signal from for example raw gyro, accelerometer, and/ormagnetometer data. Sensor fusion can be implemented in ICs such as theInvenSense MPU 3000.

In some embodiments the handheld device comprises at least oneuser-interface element other than the at least one motion sensor.

In some embodiments the at least one other user-interface element is atouch-sensitive element.

In some embodiments the at least one other user-interface element is abutton.

In some embodiments the at least one other user-interface element is ascroll-wheel.

In some embodiments, user interface functionality is provided throughadditional elements on the device. Thus these additional elements canfor example be buttons, scroll wheels, touch-sensitive fields, proximitysensors and/or the like.

The additional user interface elements can be exploited or utilized in aworkflow suitable for the field of application of the device. Theworkflow may be implemented in some user software application that mayalso control the display and thus the view represented thereon. A giveninterface element can supply multiple user inputs to the software. Forexample, a button can provide both a single click and a double click.For example, a double click can mean to advance to a subsequent step ina workflow. For the example of intraoral scanning, three steps withinthe workflow can be to scan the lower mouth, the upper mouth, and thebite. A touch-sensitive field can provide strokes in multiple directionseach with a different effect, etc. Providing multiple user inputs from auser interface elements is advantageous because the number of userinterface elements on the device can be reduced relative to a situationwhere each user interface element only provides one user input.

The motion sensors can also be exploited in a workflow. For example,lifting the device, which can be sensed by an accelerometer, canrepresent some type of user input, for example to start some action. Ina device that is a scanner, it may start scanning. Conversely, placingthe device back in some sort of holder, which can be sensed by anaccelerometer as no acceleration occur over some period of time, canstop said action.

If the action performed by the device is some kind of recording, forexample scanning, for example 3D scanning, the results of the recordingcan also be exploited as user inputs, possibly along with user inputsfrom other user interface elements. For example, with a 3D scanner witha limited depth of field, it may be possible to detect whether anyobjects within the 3D environments are present in the volumecorresponding to this depth of field by detecting whether any 3D pointsare recorded. User inputs can depend on such detected presence. Forexample, a button click on an intraoral scanner can provide a differentuser input depending on whether the scanner is in the mouth, where teethare detectable, or significantly away from and outside the mouth. Alsothe effect of motion sensor signals can be interpreted differently foreither situation. For example, the scanner may only change the viewrepresented on the display when it is outside the mouth.

In some embodiments the handheld device is adapted to change a viewingangle with which the 3D environment is represented on the at least onedisplay.

In some embodiments the handheld device is adapted to change amagnification factor with which the 3D environment is represented on theat least one display.

In some embodiments the handheld device is adapted to change a viewingposition with which the 3D environment is represented on the at leastone display.

In some embodiments the view of the 3D environment comprises a viewingangle, a magnification factor, and/or a viewing position.

In some embodiments the view of the 3D environment comprises renderingof texture and/or shading.

In some embodiments the at least one display is divided into multipleregions, each showing the 3D environment with a different view.

Thus in some embodiments the user interface functionality compriseschanging the view with which the 3D environment is displayed. Changes inview can comprise changes in viewing angle, viewing position,magnification and/or the like. A change in viewing angle can naturallybe effected by rotating the device. Rotation is naturally sensed by theaid of gyros and/or relative to gravity sensed by an accelerometer.Zooming, i.e. a change in magnification, can for example be achieved bypushing the handheld device forward and backward, respectively. Atranslational change of the viewing position, i.e., panning, can forexample be achieved by pushing the handheld device up/down and/orsideways.

In some embodiments the user interface functionality comprises selectingor choosing items on a display or any other functionality provided bygraphical user interfaces in computers known in the art. The operatormay perform the selection. The Lava C.O.S scanner marketed by 3M ESPEhas additional buttons on the handheld device, but it is not possible tomanipulate the view by these. Their only purpose is to allow navigationthrough a menu system, and to start/stop scanning.

In some embodiments the user interface functionality comprisesmanipulating the 3D environment displayed on the screen. For example,the operator may effect deformations or change the position ororientation of objects in the 3D environment. Thus, in some embodimentsthe user interface functionality comprises virtual user interfacefunctionality, which can be that the 3D data are manipulated, but thephysical 3D environment in which the device operates may not bemanipulated.

In some embodiments the handheld device is an intraoral scanner and/oran in-the-ear scanner. If the scanner comprises a tip, this tip may beexchanged whereby the scanner can become suitable for scanning in themouth or in the ear. Since the ear is a smaller cavity than the mouth,the tip for fitting into an ear may be smaller than a tip for fitting inthe mouth.

In some embodiments the handheld device is a surgical instrument. Insome embodiments, the surgical instrument comprises at least one motionsensor, which is built-in in the instrument.

In some embodiments the handheld device is a mechanical tool. In someembodiments, the tool has at least one motion sensor built in. In otherembodiments, other user-interface elements are built in as well, forexample buttons, scroll wheels, touch-sensitive fields, or proximitysensors.

In some embodiment the 3D geometry of the 3D environment is knowna-priori or a 3D representation of the environment is known a priori,i.e. before the actions (s) are performed. For example in surgery, a CTscan may have been taken before the surgical procedure. The handhelddevice in this example could be a surgical instrument that a physicianneeds to apply in the proper 3D position. To make sure this properposition is reached, it could be beneficial to view the 3D environmentfrom multiple perspectives interactively, i.e. without having to releasethe surgical instrument.

An advantage of the system, also in the above surgery example, is theability of the handheld device to record the 3D environment at leastpartially, typically in a 3D field-of-view that is smaller than thevolume represented in the a-priori data. The 3D data recorded by thehandheld device can be registered in real time with the a-priori data,such that the position and orientation of the device can be detected.

In some embodiments the 3D geometry comprises a 3D surface of theenvironment.

In some embodiments the 3D geometry comprises a 3D volumetricrepresentation of the environment.

Thus the 3D environment can be displayed as volumetric data, or assurface, or a combination thereof. Volumetric data are typicallyrepresented by voxels. Voxels can comprise multiple scalar values.Surface data are typically represented as meshed, such as triangulatedmeshes, or point clouds.

The scanning may be performed by means of LED scanning, laser lightscanning, white light scanning, X-ray scanning, and/or CT scanning.

The present invention relates to different aspects including the systemdescribed above and in the following, and corresponding systems,methods, devices, uses, and/or product means, each yielding one or moreof the benefits and advantages described in connection with the firstmentioned aspect, and each having one or more embodiments correspondingto the embodiments described in connection with the first mentionedaspect and/or disclosed in the appended claims.

In particular, disclosed herein is a method of interaction between ahandheld device and at least one display, where the method comprises thesteps of:

-   -   performing at least one action in a physical 3D environment by        means of the handheld device;    -   visually representing the physical 3D environment by the at        least one display; and    -   remotely controlling the view of the represented 3D environment        on the display by means of the handheld device.

Furthermore, the invention relates to a computer program productcomprising program code means for causing a data processing system toperform the method according to any of the embodiments, when saidprogram code means are executed on the data processing system, and acomputer program product, comprising a computer-readable medium havingstored there on the program code means.

According to another aspect, disclosed is a system comprising a handhelddevice for operating in a 3D environment and at least one display forvisualizing said environment, where the display is adapted to representsaid environment from multiple perspectives,

where said device is adapted to be held in one hand by an operator, andwhere the perspective represented in the at least one display is atleast partly determined by the motion of the operator's hand holdingsaid device.

According to another aspect, disclosed is a system comprising a handhelddevice for operating in a 3D environment and at least one display forvisualizing said environment, where the display is adapted to representsaid environment in multiple views,

where said device is adapted to be held in one hand by an operator,where the view represented in the at least one display is at leastpartly determined by the motion of the operator's hand holding saiddevice, and where the device has at least one touch-sensitive userinterface element.

The motion of the operator's hand is typically determined by a motionsensor arranged in the handheld device.

DEFINITIONS

3D geometry: A constellation of matter or its virtual representation ina three-dimensional space.

3D environment: A constellation of physical objects each having a 3Dgeometry in a three-dimensional space.

View: The way a 3D environment is represented on a display.Geometrically, a view is determined by the virtual observer's/camera'sposition and orientation. If the display is two-dimensional, the view isalso determined by the type of projection. A view may also be determinedby a magnification factor. Graphically, a view can show the 3Denvironment by means of photographs or as some kind of virtualrepresentation such as a computer graphic, or similar. A computergraphic can be rendered for example with texture and/or shading and/orvirtual light sources and/or light models for surface properties. Acomputer graphic can also be a simplified representation of the 3Denvironment, for example a mesh, an outline, or an otherwise simplifiedrepresentation. All or parts of the 3D environment can also be renderedwith some degree of transparency. A view may represent the 3Denvironment in total or only parts thereof.

Functionality: A Purpose or Intended Use.

Performing action(s) or operating functionality: Actions orfunctionality that includes some type of interaction with a 3Denvironment, such as measuring, modifying, manipulating, recording,touching, sensing, scanning, moving, transforming, cutting, welding,chemically treating, cleaning, etc. The term “operating” is thus notdirected to surgical procedures, but operating may comprise surgicalprocedures.

User Interface Functionality: Functionality for interaction between ahuman user and a machine with a display.

Handheld device: An object that has at least one functionality and thatis held by a human operator's hand or both hands while performing thisat least one functionality.

3D scanner: A device that analyzes a real-world object or 3D environmentto collect data on its shape and possibly its appearance.

Coverage of scan: The degree to which a physical surface is representedby recorded data after a scanning operation.

Motion sensor: A sensor detecting motion. Motion can be detected by:sound (acoustic sensors), opacity (optical and infrared sensors andvideo image processors), geomagnetism (magnetic sensors, magnetometers),reflection of transmitted energy (infrared laser radar, ultrasonicsensors, and microwave radar sensors), electromagnetic induction(inductive-loop detectors), and vibration (triboelectric, seismic, andinertia-switch sensors). MEMS accelerometers, gyros, and magnetometersare examples of motions sensors.

Workflow: a sequence of tasks implemented in software.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional objects, features and advantages of thepresent invention, will be further elucidated by the followingillustrative and non-limiting detailed description of embodiments of thepresent invention, with reference to the appended drawings, wherein:

FIG. 1 shows an example of the system comprising a handheld device and adisplay.

FIG. 2 shows an example of user interface functionality in the form ofremote controlling using the handheld device.

FIG. 3 shows an example of the handheld device.

FIG. 4 shows an example of a flow-chart of a method of interactionbetween a handheld device and a display.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingfigures, which show by way of illustration how the invention may bepracticed.

FIG. 1 shows an example of the system comprising a handheld device and adisplay. The handheld device 100 is in this example an intraoral dentalscanner, which records the 3D geometry of the patient's teeth. Theoperator 102 moves the scanner along the teeth of the patient 104 forcapturing the 3D geometry of the relevant teeth, e.g. all teeth. Thescanner comprises motion sensors (not visible) for taken the movement ofthe scanner into account while creating the 3D model 105 of the scannedteeth. The display 101 is in this example a PC screen displaying thedata recorded by the scanner.

FIG. 2 shows an example of user interface functionality in the form ofremote controlling using the handheld device. The motion sensors (notshown) in the handheld device 100, e.g. scanner, provide that the user102 can determine the view shown on the display 101, e.g. screen, bymoving the handheld device 100.

FIG. 2a ) shows that pointing the device 100 down can provide that the3D model 105 of the scanned teeth is shown from a downward viewingangle.

FIG. 2b ) shows that holding the scanner in a horizontal position canprovide that the viewing angle is likewise horizontally from the front,such that the 3D model 105 of the scanned teeth is shown from the front.

FIG. 3 shows an example of the handheld device.

The handheld device 100 is in this example an intraoral scanner with apistol-grip. The scanner comprises a housing 106 comprising thepistol-grip part 107, and a tip 108 adapted for insertion in the mouthof the patient. In this example the scanner also is equipped with abutton 103 which is an additional element providing user-interfacefunctionality.

The example system as shown in FIG. 1, FIG. 2 and FIG. 3 comprises adevice 100 which is a handheld intraoral scanner and a display 101 whichis a computer screen. The operator 102 may be a dentist, an assistantand/or the like. In an example, the action performance or operationfunctionality of the device 100 is to record some intraoral 3D geometry,and the user interface functionality is to rotate, pan, and zoom the 3Dmodel 105 of the scanned data on the computer screen 101. Theintegration of the user interface functionality in the device 100 isprovided by motion sensors (not visible), which can be accelerometersinside the scanner 100, whose readings determine the orientation, asseen in FIGS. 2a and 2b , of the display on the screen of the 3D model105 of the teeth acquired by the scanner 100. Additional functionality,e.g. to start/stop scanning, may be provided by the button 103 as seenin FIG. 3. In the example system, the button 103 is located where theuser's index finger can reach it conveniently.

In FIG. 1 the dentist 102 uses two hands to hold the intraoral scanner100 while scanning, but it is understood that the scanner 100 can alsobe held with one hand while scanning. The device 100 can also be heldwith one or two hands, while changing the perspective of the 3D model105 in the display 101. The example shown in FIG. 1 thus illustrates theadvantage of the touchless user interface functionality, because in manyclinical situations, the operator 102 should maintain both hands clean,disinfected, or even sterile.

The 3D user interface functionality may be exploited in a separatelocation than the operation functionality. For the above intraoralscanning system example, the scanning operation is performed in the oralcavity of the patient, see FIG. 1, while the user interfacefunctionality is more flexibly exploited when the scanner is outside thepatient's mouth, see FIGS. 2 and 3.

FIG. 4 shows an example of a flow-chart of a method of interactionbetween a handheld device and a display.

In step 101 at least one action in a physical 3D environment isperformed by means of the handheld device. This action may the scanningof teeth as shown in FIG. 1.

In step 102 the physical 3D environment is visually represented by theat least one display. This may be the display of the 3D model of thescanned teeth as seen in FIG. 1.

In step 103 the view of the represented 3D environment shown on thedisplay is remotely controlled on the display by means of the handhelddevice. This may be the control of the viewing angle of the 3D model asseen in FIG. 2.

All the steps of the method may be repeated one or more times. The orderin which the steps are performed may be different than the orderdescribed above, which is indicated by the dotted lines in the figure.If one or more of the steps are performed more times, the order of thesteps may also be different.

Although some embodiments have been described and shown in detail, theinvention is not restricted to them, but may also be embodied in otherways within the scope of the subject matter defined in the followingclaims. In particular, it is to be understood that other embodiments maybe utilised and structural and functional modifications may be madewithout departing from the scope of the present invention.

In device claims enumerating several means, several of these means canbe embodied by one and the same item of hardware. The mere fact thatcertain measures are recited in mutually different dependent claims ordescribed in different embodiments does not indicate that a combinationof these measures cannot be used to advantage.

It should be emphasized that the term “comprises/comprising” when usedin this specification is taken to specify the presence of statedfeatures, integers, steps or components but does not preclude thepresence or addition of one or more other features, integers, steps,components or groups thereof.

The features of the method described above and in the following may beimplemented in software and carried out on a data processing system orother processing means caused by the execution of computer-executableinstructions. The instructions may be program code means loaded in amemory, such as a RAM, from a storage medium or from another computervia a computer network. Alternatively, the described features may beimplemented by hardwired circuitry instead of software or in combinationwith software.

LITERATURE

-   C. Graetzel, T. Fong, S. Grange, and C. Baur. A Non-Contact Mouse    for Surgeon-Computer Interaction. Technology and Health Care, 12(3),    2004.-   Vogt S., Khamene A., Niemann H., Sauer F., An AR system with    intuitive user interface for manipulation and visualization of 3D    medical data, Stud. Health Technol. Inform. 2004; 98, pp. 397-403.

EMBODIMENTS

The following embodiments relates to one aspect of the system asdisclosed by the description herein.

1. A system comprising a handheld device and at least one display, wherethe handheld device is adapted for performing at least one action in aphysical 3D environment, where the at least one display is adapted forvisually representing the physical 3D environment, and where thehandheld device is adapted for remotely controlling the view with whichthe 3D environment is represented on the display.

2. The system according to any one or more of the preceding embodiments,wherein the view is defined as viewing angle and/or viewing position.

3. The system according to any one or more of the preceding embodiments,wherein the handheld device is adapted for remotely controlling themagnification with which the 3D environment is represented on thedisplay.

4. The system according to any one or more of the preceding embodiments,wherein the handheld device is adapted for changing the rendering of the3D environment on the display.

5. The system according to any one or more of the preceding embodiments,wherein the at least one action comprises one or more of:

-   -   measuring,    -   recording,    -   scanning,    -   manipulating, and/or    -   modifying.

6. The system according to any one or more of the preceding embodiments,wherein the 3D environment comprises one or more 3D objects.

7. The system according to any one or more of the preceding embodiments,wherein the handheld device is adapted to be held in one hand by anoperator.

8. The system according to any one or more of the preceding embodiments,wherein the display is adapted to represent the 3D environment frommultiple views.

9. The system according to any one or more of the preceding embodiments,wherein the view of the 3D environment represented in the at least onedisplay is at least partly determined by the motion of the operator'shand holding said device.

10. The system according to any one or more of the precedingembodiments, wherein the magnification represented in the at least onedisplay is at least partly determined by the motion of the operator'shand holding said device.

11. The system according to any one or more of the precedingembodiments, wherein the handheld device is adapted to record the 3Dgeometry of the 3D environment.

12. The system according to any one or more of the precedingembodiments, wherein the 3D geometry of the 3D environment is knowna-priori.

13. The system according to any one or more of the precedingembodiments, wherein the handheld device comprises at least oneuser-interface element.

14. The system according to any one or more of the precedingembodiments, wherein the at least one user-interface element is at leastone motion sensor.

15. The system according to any one or more of the precedingembodiments, wherein the handheld device comprises at least two motionsensors providing sensor fusion.

16. The system according to any one or more of the precedingembodiments, wherein the user interface functionality comprises the useof gestures.

17. The system according to any one or more of the precedingembodiments, wherein the gestures are detected by the at least onemotion sensor.

18. The system according to any one or more of the precedingembodiments, wherein the handheld device comprises at least oneuser-interface element other than the at least one motion sensor.

19. The system according to any one or more of the precedingembodiments, wherein the at least one other user-interface element is atouch-sensitive element.

20. The system according to any one or more of the precedingembodiments, wherein the at least one other user-interface element is abutton.

21. The system according to any one or more of the precedingembodiments, wherein the at least one other user-interface element is ascroll wheel.

22. The system according to any one or more of the precedingembodiments, wherein the handheld device is adapted to change a viewingangle with which the 3D environment is represented on the at least onedisplay.

23. The system according to any of the preceding embodiments, whereinthe handheld device is adapted to change a magnification factor withwhich the 3D environment is represented on the at least one display.

24. The system according to any one or more of the precedingembodiments, wherein the handheld device is adapted to change a viewingposition with which the 3D environment is represented on the at leastone display.

25. The system according to any one or more of the precedingembodiments, wherein the view of the 3D environment comprises a viewingangle, a magnification factor, and/or a viewing position.

26. The system according to any one or more of the precedingembodiments, wherein the view of the 3D environment comprises renderingof texture and/or shading.

27. The system according to any one or more of the precedingembodiments, wherein the at least one display is divided into multipleregions, each showing the 3D environment with a different view.

28. The system according to any one or more of the precedingembodiments, wherein the 3D geometry comprises a 3D surface of theenvironment.

29. The system according to any one or more of the precedingembodiments, wherein the 3D geometry comprises a 3D volumetricrepresentation of the environment.

30. The system according to any one or more of the precedingembodiments, wherein the handheld device is an intra-oral 3D scanner.

31. The system according to any one or more of the precedingembodiments, wherein the handheld device is a surgical instrument.

32. The system according to any one or more of the precedingembodiments, wherein the handheld device is a mechanical tool.

33. The system according to any one or more of the precedingembodiments, wherein the handheld device is an in-ear 3D scanner.

34. The system according to any one or more of the precedingembodiments, wherein the at least one display is arranged separate fromthe handheld device.

35. The system according to any one or more of the precedingembodiments, wherein the at least one display is arranged on a cart.

36. The system according to any one or more of the precedingembodiments, wherein the at least one display is defined as a firstdisplay, and where the system further comprises a second display.

37. The system according to any one or more of the precedingembodiments, wherein the second display is arranged on the handhelddevice.

38. The system according to any one or more of the precedingembodiments, wherein the second display is arranged on the handhelddevice in a position such that the display is adapted to be viewed bythe operator, while the operator is operating the handheld device.

39. The system according to any one or more of the precedingembodiments, wherein the second display indicates where the handhelddevice is positioned relative to the 3D environment.

40. The system according to any one or more of the precedingembodiments, wherein the first display and/or the second displayprovides instructions for the operator.

41. The system according to any one or more of the precedingembodiments, wherein visual information is provided to the operator onone or more means other than the first display.

42. The system according to any one or more of the precedingembodiments, wherein audible information to the operator is provided tothe operator.

43. The system according to any one or more of the precedingembodiments, wherein the scanning is performed by means of LED scanning,laser light scanning, white light scanning, X-ray scanning, and/or CTscanning.

44. A method of interaction between a handheld device and at least onedisplay, where the method comprises the steps of:

-   -   performing at least one action in a physical 3D environment by        means of the handheld device;    -   visually representing the physical 3D environment by the at        least one display; and    -   remotely controlling the view of the represented 3D environment        on the display by means of the handheld device.

45. A computer program product comprising program code means for causinga data processing system to perform the method of any one or more of thepreceding embodiments, when said program code means are executed on thedata processing system.

46. A computer program product according to the previous embodiment,comprising a computer-readable medium having stored there on the programcode means.

The invention claimed is:
 1. A scanning system for scanning a 3Denvironment, the scanning system comprising: a handheld device includingan optical scanner, wherein the 3D environment to be scanned is selectedby pointing the optical scanner at the 3D environment; and at least onedisplay remotely connected to the handheld device, wherein the handhelddevice is adapted for performing at least one scanning action in aphysical 3D environment, and the at least one display is adapted forvisually representing the physical 3D environment; and the handhelddevice includes a user interface for remotely controlling the display toadjust the view with which the 3D environment is represented on thedisplay.
 2. A system according to claim 1, wherein the handheld deviceis adapted to record the 3D geometry of the 3D environment.
 3. A systemaccording to claim 1, wherein the user interface includes means formanually switching between performing the at least one scanning actionand remotely controlling the view.
 4. The system according to claim 1,wherein the handheld device comprises at least one motion sensor.
 5. Thesystem according to claim 4, wherein the view of the 3D environmentrepresented in the at least one display is at least partly determined bythe at least one motion sensor.
 6. The system according to claim 4,wherein functionality of the user interface comprises a use of gestures.7. The system according to claim 6, wherein the gestures are detected bythe at least one motion sensor.
 8. The system according to claim 4,wherein the user-interface is other than the at least one motion sensor.9. The system according to claim 1, wherein the handheld device isadapted to change a viewing angle with which the 3D environment isrepresented on the at least one display.
 10. The system according toclaim 1, wherein the handheld device is adapted to change amagnification factor with which the 3D environment is represented on theat least one display.
 11. The system according to claim 1, wherein thehandheld device is an intra-oral 3D scanner.
 12. The system according toclaim 1, wherein the handheld device includes a surgical instrument. 13.The system according to claim 1, wherein the handheld device includes amechanical tool.
 14. The system according to claim 1, wherein thehandheld device is an in-ear 3D scanner.
 15. The system according toclaim 1, wherein the at least one display is defined as a first display,and where the system further comprises a second display.
 16. The systemaccording to claim 15, wherein the second display indicates where thehandheld device is positioned relative to the 3D environment.
 17. Thesystem according to claim 15, wherein the first display and/or thesecond display provides instructions for the operator.
 18. The systemaccording to claim 1, wherein audible information is provided to theoperator.
 19. A system comprising: a handheld device and at least onedisplay; wherein the handheld device is adapted for switching betweenperforming at least one action in a physical 3D environment, wherein theat least one display is adapted for visually representing the physical3D environment; and remotely controlling the display to adjust the viewwith which the 3D environment is represented on the display; wherein thehandheld device is an intra-oral 3D scanner and the at least one actionperformed in the physical 3D environment is scanning and that the viewis remotely controlled by at least one motion sensor arranged in thehandheld device, and wherein an actuator provided on the handheld deviceswitches between performing the at least one action and remotelycontrolling the view.